What’s Copyright?

• Web scraping research is limited by a patchwork of copyright laws across different jurisdictions.
• Text, images, videos, music, databases and other materials used by researchers are subject to copyright law, which gives the owner (e.g., businesses, individuals) the right to prohibit exploitation (e.g., replication or capturing) without permission.
• The fact that data is available on the internet does not imply that anyone can extract and store the information^[6].
• Although some publishers provide limited copyright licences for TDM use, paid licencing is not an affordable option for many critical research projects^[3].

What is NOT Copyright?

• Copyright does not protect ideas, concepts, systems, or methods of doing something^[7].
• While certain activities such as research, teaching, parody, critique, news reporting, or commentary may be exempt from copyright law, each potential infringement must be evaluated on a case-by-case basis, as copyright issues are not always unambiguous like personal data issues.
• Extracting only factual information, such as product names and prices, may be permissible without infringing copyright laws, as facts are typically not protected by such laws^[6].

Research Exceptions to Copyright

US: Digital Millennium Copyright Act

• Scraping facts is allowed^[8].
• Fair Use Doctrine:
  • purpose and character of the use matters, including whether the use is of a commercial nature or is for nonprofit educational purposes;
  • nature of the copyrighted work - the degree to which the work that was used relates to copyright’s purpose of encouraging creative expression;
  • amount and substantiality of the portion used in relation to the copyrighted work as a whole;
  • effect of the use upon the potential market for or value of the copyrighted work^[9].
• Limited database protection^[8].

EU: Copyright Directive

• Scraping facts is allowed but may vary in country specific law.
• Copyright exceptions are country specific.
• There is no ‘fair use’ doctrine in EU copyright law comparable to that of the US. Instead, EU law provides an explicit list of exceptions from copyrights granted to the rights holders, each with a specific scope^[10].
• Broad database protection based on Database Directive^[8].

Database Rights

Database rights are a subset of copyright. A database is an organized collection of materials that allows users to search and access individual pieces of information.

Copyright law protects databases when the way the data is selected or arranged is original and creative. Therefore, scraping cannot result in copying and, for example, republishing the original database’s structure (or a substantial part of it)^[11].

Non-original databases can also be protected if a significant investment was made in obtaining, verifying, and presenting the data^[12].

When scraping a data source that may be subject to database rights, consider:

• scraping only some data,
• scraping only the data itself (without the replication of the organization of that data),
• limiting the data scraped to factual or non-copyrighted data^[6].

The TDM Exception

• The new “Digital Single Market” (DSM) directive on copyright permits scraping (reproduction and extraction) of data from databases for the purposes of text and data mining, even if the data is protected by copyright.
• However, the exception is limited: database owners have the option to restrict the reproduction and extraction of databases and their content.
• That restriction must be implemented in such a way that bots and crawlers, among others, can see it (therefore, on a website there should be installed for example a special program communicating visiting scraping programs that scraping is prohibited).
• Therefore, unless you take a large amount of data/structure and later republish or sell it, there is a good chance you will not violate any intellectual property rights^[11].

How are Legal Terms Enacted?

The response is based on the kind of legal terms of service that are in place:

• clickwrap: terms are considered clickwrap agreements where the user is required to take an action by which he confirms the consent to the ToS;
• browsewrap: terms are considered browsewrap agreements if they are posted on a website, typically via a link at the bottom of the page;
• scrollwrap: sometimes seen as a sub-category of clickwrap, a scrollwrap agreement requires the users to scroll through the text of the ToS before they get to the “I Agree” button;
• sign-in-wrap: sign-in-wrap agreements are a hybrid of browsewrap and clickwrap. The act of acceptance here is the signing in or logging in with a note stating that by doing so, you are agreeing to the ToS^[13].

Clickwrap vs Browsewrap

What is Personal Data?

• Personal data, or personally identifiable information (PII), refers to any data that can directly or indirectly identify an individual.
• Common types of personal data include name, email, phone number, address, user name, IP address, date of birth, employment information, bank/credit card information, medical data, and biometric data.
• Public personal data is not an exception so analysis must be conducted before scraping it.
• Different legal jurisdictions (US, EU, Canada, Australia etc.) have different regulations for personal data so you need to identify the jurisdiction in which the data owners reside^[6].

GDPR Compliance

• The General Data Protection Regulation (GDPR) covers the European Economic Area (EEA).
• Although it was drafted and passed by the European Union (EU), it imposes obligations on all organisations that target or gather information about EU citizens^[14].
• To use or store personal data of EU citizens, a company or an organisation must comply with one or more of the legal reasons described by GDPR. You may, therefore, need either to:
  • receive a consent of a person of which data you’re going to process,
  • be performing a task carried out in the public interest or, as a controller or even as a third party,
  • have a legitimate interest to process the data and such processing is necessary to achieve that interest^[11].

Lawful Reasons (GDPR)

• Contract - an organisation has a contract with the data subject (person whose data we have), for example, a contract to supply goods or services or an employment contract.
• Legal Obligation - an organisation is fulfilling a legal obligation, such as when data processing is required by law.
• Public Task - to complete a public task, mostly relating to the tasks of public administrations such as schools, hospitals, and municipalities.
• Vital Interest - when data processing is in the data subject’s vital interests, for example, when this might protect their life.
• Legitimate Interest - for example, when a bank uses personal data to determine whether the data subject is eligible for a higher-interest savings account.

In all other cases, the company or organisation must obtain the data subject’s permission (known as “consent”) before collecting or reusing their personal information^[15].

Type of Personal Data (GDPR)

Next to establishing the lawful reason for scraping data, you should also consider the type of personal data being collected. Sensitive data is a subject to additional rules and requires explicit consent to be given for this data to be scraped and stored.

Sensitive data includes:

• racial or ethnic origins,
• genetic data,
• political opinions,
• biometric data that can uniquely identifying someone,
• religious or philosophical beliefs,
• health, sex life or sexual orientation data,
• or trade union membership.

Therefore, you should avoid scraping this data unless you have explicit consent and legitimate reason to do so^[16].

Data Protection Principles (GDPR)

If you process data, you have to do so according to the data protection principles:

• Lawfulness, fairness and transparency - Processing must be lawful, fair, and transparent to the data subject.
• Purpose limitation - You must process data for the legitimate purposes specified explicitly to the data subject when you collected it.
• Data minimization - You should collect and process only as much data as absolutely necessary for the purposes specified.
• Accuracy - You must keep personal data accurate and up to date.
• Storage limitation - You may only store personally identifying data for as long as necessary for the specified purpose.
• Integrity and confidentiality - Processing must be done in such a way as to ensure appropriate security, integrity, and confidentiality (e.g. by using encryption).
• Accountability - The data controller is responsible for being able to demonstrate GDPR compliance with all of these principles^[14].

Data Policies (GDPR)

Even when you received an explicit consent from the data subject, you need to ensure that the correct data retention and access policies are in place:

• Ensure that data subjects are aware of the company’s data protection and privacy policy.
• Comply with Data Subject Access Rights (DSAR), including the right to withdraw consent, request a copy of data, or request deletion of data.
• If consent is withdrawn or a DSAR for deletion is received, delete or anonymize the personal data as it is no longer legally justifiable to retain it^[16].

Residential IPs (GDPR)

Residential proxies provide real IP addresses of actual devices. When using a residential IP for scraping (or even just accessing web pages), you appear to be accessing websites and social media platforms from an actual home-based IP^[17].

• IP addresses are considered personally identifiable information (PII) under the GDPR regulation.
• Ensure that any EU residential IPs used as proxies are GDPR compliant.
• Obtain explicit consent from the owner of a residential IP before using it as a web scraping proxy.
• If obtaining residential proxies from a third-party provider, ensure that they have obtained consent and are in compliance with GDPR before using the proxy^[16].